LDPC Code Design for Transmission of Correlated 
Sources Across Noisy Channels Without CSIT 

Arvind Yedla, Henry D. Pfister, and Krishna R. Narayanan 

Department of Electrical and Computer Engineering 

Texas A&M University 

Email: {yarvind,hpfister,krn} @tamu.edu 



o 



C/5 



m 

On 
O 

rS 

o 
o 

> 

X 



Abstract — We consider the problem of transmitting correlated 
data after independent encoding to a central receiver through 
orthogonal channels. We assume that the channel state informa- 
tion is not known at the transmitter. The receiver has access to 
both the source correlation and the channel state information. 
We provide a generic framework for analyzing the performance 
of joint iterative decoding, using density evolution. Using dif- 
ferential evolution, we design punctured systematic LDPC codes 
to maximize the region of achievable channel conditions, with 
joint iterative decoding. The main contribution of this paper is 
to demonstrate that properly designed LDPC can perform well 
simultaneously over a wide range of channel parameters. 

Index Terms — LDPC codes, density evolution, correlated 
sources, non-systematic encoders, joint decoding, differential 
evolution. 

I. Problem Setup 

Consider the problem of transmitting the outputs of two 
discrete memoryless correlated sources, (C7i, t/2), to a cen- 
tral receiver through two independent discrete memoryless 
channels with capacities C\ and C2, respectively. The system 
model is shown in Figure 1. We will assume that the channels 
belong to the same channel family, and that each channel can 
be parametrized by a single parameter a (e.g., the erasure 
probability for erasure channels). The two encoders are not 
allowed to communicate. Hence they must use independent 
encoding functions, which map k input symbols (Ui and U2) 
to n\ and n 2 output symbols (Xi and X2), respectively. The 
rates of the encoders are given by R\ = fc/m and _R 2 = fc/n 2 . 
The decoder receives (Yi,Y 2 ) and makes an estimate of 
(Ui,U2). This joint source-channel coding problem can be 
seen to be an instance of Slepian-Wolf coding [1] in the 
presence of a noisy channel. 
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Figure 1. System Model 



over a large set of channel parameters. Therefore, by the 
symmetry of the problem, we assume that both the encoders 
use identical codes of rate R (i.e., R — k/n, n\ — n-i — ri). 
Reliable transmission over a channel pair (0:1,0:2) is possible 
as long as the Slepian-Wolf conditions (1) are satisfied. 
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For a given pair of encoding functions of rate R and a joint 
decoding algorithm, a pair of channel parameters (0:1,0:2) is 
achievable if the encoder/decoder combination can achieve 
an arbitrarily low probability of error for limiting block- 
lengths (k — *• 00). We define the achievable channel parameter 
region (ACPR) as the set of all channel parameters which 
are achievable. Note that the ACPR is the set of all channel 
parameters for which successful recovery of the sources is 
possible for a fixed encoding rate pair (R, R). We also define 
the Slepian-Wolf region as the set of all channel parameters 
(o 1; o 2 ) for which (1) is satisfied. The Slepian-Wolf region for 
the erasure channel family is shown in Figure 2. We wish to 
find LDPC codes which result in large ACPRs, preferably the 
entire Slepian-Wolf region. Coding schemes which have large 
ACPRs are desirable because, for example, such a scheme can 
minimize the outage probability for non-ergodic channels. 
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The problem we consider is to design a graph based code, 

for Which a joint iterative decoder can successfully decode *jf « 2 - The Slepian-Wolf region for erasure channels, for a fixed rate pair 

(it, H) 
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In this paper, we consider the following scenarios: 

1) The channels are additive white Gaussian noise (AWGN) 
channels and the source correlation is modeled through 
a virtual correlation channel analogous to a binary 
symmetric channel (BSC). 

2) The channels are erasure channels and the source corre- 
lation is modeled through erasures. 

II. Background 

A. Prior Work 

Recently this problem has attracted attention in the area 
of wireless sensor networks. It is a specific case of the 
sensor reachback problem [2], Although separation between 
source and channel coding is known to be optimal for AWGN 
channels [2], it is desirable to take a joint source-channel 
coding approach (via direct channel coding and joint decoding 
at the receiver) [3], When the channel parameters are known 
a priori at the transmitter, graph based coding schemes have 
been proposed [4], [5], which provide near optimal perfor- 
mance under joint iterative decoding. [3], [6] discuss the 
performance of concatenated LDGM codes under symmetric 
channel conditions. In [7], the authors study the performance 
of Turbo codes over varying channel conditions. Even when 
the channel parameters are known at the transmitter, it can be 
hard to design codes that perform well for symmetric channel 
conditions [8]. For the case of erasure correlated sources, our 
previous work gives a provable capacity achieving sequence 
of codes for the erasure channel under symmetric channel 
conditions [9]. 

However, in several practical situations, it is unrealistic for 
the transmitters to have a priori knowledge of the channel 
parameters. A single code now needs to perform well over the 
entire Slepian-Wolf region, making the problem of designing a 
good code even harder. In [7], [8], this problem is considered; 
however, the authors choose a code that performs well at one 
point on the Slepian-Wolf region and evaluate its performance 
for different channel parameters. As a result, the performance 
of the code is far from the optimal performance for some 
channel parameters. In this paper, we will show that by 
choosing an appropriate low density parity check (LDPC) code 
ensemble and optimizing it to perform well at several points 
on the Slepian-Wolf region, significantly better performance 
can be obtained. 

B. The Slepian-Wolf Region 

Let I ai (Xi;Yi) (I a2 {X 2 ;Y 2 )) denote the mutual informa- 
tion between X x and Y\ (X 2 and Y 2 ) when the underlying joint 
distribution is parametrized by oi\ (a 2 ). The following theorem 
shows the existence of codes which have large ACPRs. 

Theorem 1: For a fixed pair of channel conditions (ai, a 2 ), 
which is not known at the transmitter, random coding with 
typical-set decoding at the receiver can achieve an average 
probability of error P e . Ql! a 2 (over all encoders with a rate R) 
bounded above by 2~ n '( ai ' a2 \ where 

r(ai,a 2 )=niin(7 ai (X 1 ;F 1 )-i? J ff(C/ 1 \U 2 ), 
I a2 {X 2 ;Y 2 )-RH{U 2 \U 1 ), 
I Ql (*i;yi) + I a2 (X2-,Y 2 ) - RH(U U U 2 )). 



Hence, there exists an encoder for which the probability of 
error 

P < r,-nr(a 1 ,a 2 ) 

1 e,cti ,ft2 — 

Proof: The proof follows from standard random coding 
arguments. It is omitted here due to space constraints. ■ 

Remark 1: A simple application of Fano's inequality shows 
that any pair of channel parameters for which r(ai,a 2 ) < 
are not achievable (the probability of error is strictly bounded 
away from zero). For binary memoryless symmetric (BMS) 
channels, the condition r(a 1} a 2 ) > translates to the condi- 
tions in (1). So, the conditions in (1) are both necessary and 
sufficient for transmission over BMS channels. 

Corollary 1: For BMS channels, the achievable channel 
parameter region for a random code is a dense subset of the 
entire Slepian-Wolf region for limiting block-lengths. 

Proof: Let S denote the Slepian-Wolf region and for n e 
N, let 
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If Pe denotes the random variable representing the prob- 
ability of error of a randomly chosen encoding map with 
block-length n. Given 5 > 0, there exists n e N such that 

n 3 2 -nr( ai ,a 2 ) < j_ jj^ 

P(P e (n) < 6 | ai,a 2 ) > 1 - P(P e (n) > n 3 2-" r ( ai '" 2 ) | a u a 2 ) 

1 

^ 1 ~-3' 

by the Markov inequality. We define the dominant exponent 
r„ = min( Ql Q2 ) G a„ Kcni, 02). Then we can upper bound the 
probability that a random code is bad for at-least one channel 
pair in A„ by a simple union bound, which gives us 
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lim P(P e (ll) > 8) = 0. 



So, for limiting block-lengths, a single randomly chosen code 
can achieve a vanishing probability of error for a dense subset 
of the entire Slepian-Wolf region. ■ 

Remark 2: Corollary 1 also holds for random linear codes. 

We conclude that, for a given rate pair (R, R), a single 
encoder/decoder pair suffices to communicate the sources over 
all pairs of BMS channels in the Slepian-Wolf region. Thus, 
one can obtain optimal performance even without knowledge 
of (ai, a 2 ) at the transmitter. We refer to such encoder/decoder 
pairs as being universal. This means that random codes with 
typical-set decoding are universal for BMS channels. 

Random coding with typical-set decoding is also universal 
with respect to channel types, since the performance of random 
coding depends only on channel capacity. In this paper, we are 
interested in universality with respect to channel parameters, 
for a particular channel type. 

While random codes with typical-set decoding are uni- 
versally good, this scheme is clearly impractical due to its 
large complexity. This motivates the search for low complexity 
encoding/decoding schemes which are universal. It was shown 
in [9] that codes that are good for the single user scenario 



need not be universal for the JSCC problem considered in 
this paper. It was conjectured in [8] that LDPC codes do not 
perform well for this problem 1 . To the contrary, we show that 
properly designed LDPC codes can have a large ACPR. 

III. Source Correlation 

In this section, we describe two possible correlation models 
between the sources. These models might appear restrictive, 
but will provide sufficient insight into the design of codes that 
perform well for arbitrary correlated sources. Our analysis in 
Section IV admits general correlation models. 

A. BSC Correlation 

Consider a symmetric correlation model which can be 
defined in terms of a single parameter. For binary sources 
Ui and U 2 , this parameter is given by Pr(Ui = U 2 ), It 
is useful to visualize this correlation by the presence of an 
auxiliary binary symmetric channel (BSC) with parameter 

1 — p between the sources. In other words, U 2 is the output 
of a BSC with input XJ\ i.e., U 2 = U\ + Z. Here Z is a 
Bernoulli-(l — p) random variable and can be thought of as 
an error. Let h 2 () denote the binary entropy function. Then, 
H{U!\U 2 ) = HiUilUt) = h 2 (p) and H(U U U 2 ) = l + h 2 (p). 

This correlation model can be incorporated into the Tanner 
graph at the decoder (described in Section IV-B) as check 
nodes between the source bits, with a hidden node representing 
the auxiliary random variable Z (which carries a constant log- 
likelihood ratio log - J=£ ) attached to the check node. For this 
scenario, the decoder does not require any side information 
i.e., it does not need to know the realization of the auxiliary 
random variable Z. 

B. Erasure Correlation 

Another common way to model the source correlation is 
through erasures. Let Z be a Bernoulli-p random variable. 
The correlation between U\ and U 2 is defined by 

{i.i.d. Bernoulli \ r.v.s,if Z = 
same Bernoulli | r.v. U , if Z = 1 

We have H(U!\U 2 ) = F(C7 2 |^i) = 1 -p and H{U U U 2 ) = 

2 — p. This correlation model can be incorporated into the 
Tanner graph at the decoder with the presence or absence 
of a check node between the source bits depending on the 
auxiliary random variable Z. Note that the decoder requires 
the realization of the random variable Z (for each of the 
source bits) as side information. This model can also be 
thought of as having two types of BSC correlation between 
the source bits, one with parameter and one with parameter 
1. The correlation parameter p determines how many bits are 
correlated with parameter 1. 

IV. Analysis 

A. Puncturing 

It is shown in [10] that correlated codes are suboptimal 
when transmitting correlated sources over independent chan- 
nels. The conditions in (1) implicitly assume the use of uncor- 
rected codes i.e., we require the average mutual information 

'The authors consider only systematic LDPC codes 



(over the code ensemble) 

I(X 1 ;X 2 )=0. 

This condition is clearly not satisfied when we use a systematic 
LDPC ensemble. This also explains the loss in performance 
of systematic LDPC codes when compared to Turbo codes, as 
shown in [8]. So, to ensure the independence of the transmitted 
codewords, we use LDPC ensembles which can be represented 
by a punctured systematic encoder. 

B. LDPC Codes 

Assume that the sequences Ui and U2 are encoded using 
LDPC codes with a degree distribution pair (\,p), with a 
punctured systematic encoder. Let the fraction of punctured 
(systematic) bits be 7. Based on standard notation [11], we 
let A(x) = ^2 t XiX 1 ^ 1 be the degree distribution (from an 
edge perspective) corresponding to the variable nodes and 
p( x ) = Si Pi x% ~ X be the degree distribution (from an edge 
perspective) of the parity-check nodes in the decoding graph. 
The coefficient A^ (resp. pi) gives the fraction of edges that 
connect to the variable nodes (resp. parity-check nodes) of de- 
gree i. Likewise, Li (resp. Ri) is the fraction of variable (resp. 
parity-check) nodes with degree i. Also, let V = {i \ \i 7^ 0} 
and V — {i \ pi ^ 0} be the support sets of the variable and 
parity-check degree distributions respectively. The rate pair of 
the two codes after puncturing is (R, R), where 



i? = i?(A,p) 



1 



1 



Jo p(x) dx s 
l\ ti\(x)d X/ 



1 



(2) 



The Tanner graph [11] for the joint decoder is shown in 
Figure 3. Code 1 corresponds to the bottom half of the graph, 
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Figure 3. Tanner Graph of an LDPC Code with source correlation 

code 2 corresponds to the top half and both the codes are 
connected by correlation nodes at the punctured bits. Let a£ 
and hi denote the density 2 of the messages emanating from the 
variable nodes at iteration £, corresponding to codes 1 and 2. 
The density evolution equations [11] can be written as follows 



a^+i — 



lf[L (KM)) + (1 - 7)3bmsc 
7 /(i(p(a,))) + (l-7)b: 



BMSC 



© A(p(a/0) 
© Mp(be)), 



(3) 



2 Assuming that the transmission alphabet is {±1}, the densities are 
conditioned on the transmission of the all one codeword (the all zero codeword 
assumption). 



where A(a) - E^a®^ 1 ), L(a) = Ei*a® (i_1) > p(a) = 
EiPi aH s 3bmsc and beivisc are the densities of the 
likelihood ratios received from the channel. The function / 
at the correlation nodes depends on the equivalent chan- 
nel corresponding to the correlation model, as described in 
[12]. For example, in the case of a BSC correlation with 
probability p, we introduce a parity-check at the correlation 
nodes which evaluates to a Bernoulli-p random variable i.e., 
/(a) = a BSC ( p ) EH a. For a BEC correlation with probability p, 
there is a parity-check at the correlation node with probability 
p and with probability 1 — p there is no parity-check, so 
/(a) = (l-p)+pa. 

Using the error functional [11] 

,o- j ,0+ 

<£(a) = / a(x)dx+- / a(x)dx, 
J-oo 2 Jo- 

die residual error probability at iteration £, (e^e^), is com- 
puted as 



= € 



7/FL (p(be))) + (1 - 7) a BMSC 
7 /(£(p(a £ )))+(l- 7 )b; 



BMSC 



®L(p{a e )) 
®L(p(bi)) 



For two residual error probabilities (ei,e2) and (ei,e2), we 
define (ei,e2) if? (ei,e2) iff ei < ei and e2 < §2. 

C. Differential Evolution 

Throughout this section, we use x to denote an element 
of R™ and Xi to denote its ith component. The correlation 
parameter p is fixed and the set of variable and parity-check 
degrees (V and V) over which we need to optimize the degree 
profiles A and p are assumed to be known. We design LDPC 
codes for this scenario using differential evolution [13], for 
a design rate Rd- Differential evolution has been shown to 
provide good results for designing LDPC codes over single 
user channels [14]. 

In an n-dimensional search space, a fixed number of vectors 
are randomly initialized and then evolved over time, explor- 
ing the search space, to locate the minima of the objective 
function. Let 



A™" 1 = {x£ 



E< 



1, Xi > 0, i = 1, 



denote the unit simplex and n v — |V|, n p — \V\. Then, the 
search space for all variable (check) degree profiles is A™" -1 
(A" p_1 ). The optimization is performed over the search space 

S = A™" -1 x A" p_1 , with parameter vectors x = [x\,x p ] 3 , 



where x\ £ A r 



£ A % 



In our optimization procedure, we expand the search space 
to S' = {x £ M n "+"*,£^A) 4 = l,Y,i{x P )i = 1}, for 
simplicity in the crossover stage. 

For the optimization to work well, differential evolution 
requires an initial population of trial vectors which are spread 
out uniformly across the search space. We generate an initial 
population of trial degree distributions by uniformly sampling 
the degree distributions from the unit simplex. To obtain 

3 (x\,V) and (x p ,V) correspond to the variable and parity node degree 
profiles respectively. 




a sample x uniformly from A n_1 , we generate uniform 
random variables m ~ <7[0, l],i = 1,2, ••• ,n — 1. Define 
Mo = 0, u n = 1 and let ir u be the permutation that sorts (uj) 
in ascending order i.e., if i < j, then «„•„(,) < u n u-\. For 
i = 1, • • • , n, define Xi — wwj) — ww^-i), and x — (xt). 
Then x has a uniform distribution over A™ -1 . 

Let C be a finite subset of channel parameters (01,02) 
that correspond to the sum rate constraint of the Slepian-Wolf 
conditions for a design rate Rd- Let Y : S' x C — >• [0, 1] x [0, 1], 
(x, 0:1,02) 1— > (ei,e2) be the function that gives the residual 
error probability 4 (using joint density evolution as described 
in Section IV-B) for each decoder, for a pair of codes with 
degree distribution x (i.e., (x\,x p )), when transmitted over 
channels with parameters (ai, 02). We use discretized density 
evolution [15] (with 9 bit linear quantization over a likelihood 
ratio range [-20, 20]). 

For our design, we want the code to achieve an arbitrarily 
low probability of error on C and we want the rate of the code 
(R(x)) to be as close to the design rate (Rd) as possible. So, 
we define the cost function, 



J r (x) = a-\ y^ (l- l{( ai: a 2 )\r(x, ai ,a 2 )<(T,T)}) 

+ b-(R d -R{x)), 

if x £ S and J-(x) = 00, if x £ S'\S. The rate of the code 
R(x) = R(x\,x p ) is computed as in (2). The constants a and 
b are chosen through trial and error. The parameters chosen 
for the designs considered in this paper are t = 10~ 5 ,a = 10 
and b = 30. The optimization is then setup as 

min J-(x). 

xes' 

We use a variant of differential evolution, with the following 
mutation and recombination scheme [14]. Parameter vectors 
^1,^2,^3 and X4 are selected at random from the current 
population. Let Xb denote the best member of the current 
population. A trial vector x is generated as 

x = xi, + 0.5 • (xi — X2 + X3 — xa). 



V. Results and Concluding Remarks 

We showed that the Slepian-Wolf conditions are necessary 
and sufficient for communication of correlated sources through 
independent BMS channels, without channel state information 
at the transmitter. This implies that a single random code is 
sufficient to communicate with vanishing probability of error, 
for the entire Slepian-Wolf region. 

We designed punctured systematic LDPC codes for the 
scenarios described in Section III. The design was performed 
to maximize the ACPR, in contrast to previous work. The 
results are given below. 

4 We set the maximum number of iterations to 100 for all the designs 
considered in this paper. Density evolution is stopped when the maximum 
number of iterations are reached or the difference in the residual error 
probability between successive iterations is less than 10 -8 . 



A. BSC Correlation 

We optimized the degree profiles for BSC correlated sources 
using the procedure described in Section IV-C, for trans- 
mission over the AWGN channel. The source correlation 
parameter was p — 0.9 and the optimization was performed 
for a design rate Rd — 0.5 after puncturing, resulting in the 
following degree profiles 

\{x) = 0.14895a; + 0.3963a; 2 + 0.13192a; 3 + 0.18799a; 6 



0.0050071a; 8 + 0.064285a; 14 + 0.065546a; 19 , 



,24 



p(x) = 0.64719a; 3 + 0.13662a; + 0.14429a; i4 + 0.071903a; 

The ACPR for this code is shown in Figure 4. Also 
shown in the figure is the Slepian-Wolf region for the 
rate pair (0.282, 0.282), which has a transmission rate 
pair (0.423,0.423) after puncturing. These results show 
that ensembles optimized using differential evolution almost 
achieve the entire Slepian-Wolf region. For a rate pair of 
(0.423, 0.423), the best (theoretical) achievable signal to noise 
ratio (SNR) for symmetric channel conditions is —2.68 dB. 
The optimized code can perform well, for symmetric channel 
conditions, up-to an SNR of —1.82 dB, which is at a gap of 
0.86 dB from the theoretical limit. 



2 
1 



§ 1 

Z -2 

-3 

-4 
































































^^ 




















\ 


















\ 














































-5 
-6 
























6 - 


5 




4 - 


3 - 
sr 


2 - 
JR, (d 


1 
B) 






3 



Figure 4. ACPR (Density Evolution threshold) of an optimized (AWGN 
channel) LDPC Code of rate 0.282 

B. Erasure Correlation 

We use the procedure described in Section IV-C to design 
LDPC codes for the erasure channel, when the sources are 
correlated with an erasure probability of p = 0.5. The 
optimization was performed for a design rate of Rj = 0.57 
after puncturing, resulting in the following degree profiles 

X(x) = 0.3633a; + 0.2834a; 2 + 0.2315a; 6 + 0.1217a; 19 , 
p{x) = 0.531776a; 3 + 0.468224a; 5 . 

The ACPR for this code is shown in Figure 5. Also 
shown in the figure is the Slepian-Wolf region for the rate 
pair (0.3308, 0.3308) which has a transmission rate pair 
(0.4962,0.4962) after puncturing. These results show that en- 
sembles optimized using differential evolution almost achieve 
the entire Slepian-Wolf region. 

C. Future Work 

The main motivation for this research was to answer the fol- 
lowing question: Do code ensembles exist which can simulta- 
neously achieve the entire Slepian-Wolf region under message 
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Figure 5. ACPR (Density Evolution threshold) of an optimized (erasure 
channel) LDPC Code of rate 0.3308 



passing decoding? This question still remains unanswered. 
Although classical LDPC codes can have a large ACPR, it is 
still not clear if they can be universal or if they are strictly sub- 
universal. There also seems to be a complexity-universality 
trade-off at the decoder, which needs to be addressed. 
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